{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Throughtput Benchmarking  Seldon-Core on GCP Kubernetes\n",
    "\n",
    "The notebook will provide a benchmarking of seldon-core for maximum throughput test. We will run a stub model and test using REST and gRPC predictions. This will provide a maximum theoretical throughtput for model deployment in the given infrastructure scenario:\n",
    "  \n",
    "   * 1 replica of the model running on n1-standard-16 GCP node\n",
    "   \n",
    "For a real model the throughput would be less. Future benchmarks will test realistic models scenarios.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Create a cluster of 4 nodes of machine type n1-standard-16. You can use GKE console or `gcloud` command line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install helm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "serviceaccount/tiller created\n",
      "clusterrolebinding.rbac.authorization.k8s.io/tiller created\n",
      "$HELM_HOME has been configured at /home/clive/.helm.\n",
      "\n",
      "Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.\n",
      "\n",
      "Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.\n",
      "To prevent this, run `helm init` with the --tiller-tls-verify flag.\n",
      "For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation\n",
      "Happy Helming!\n"
     ]
    }
   ],
   "source": [
    "!kubectl -n kube-system create sa tiller\n",
    "!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller\n",
    "!helm init --service-account tiller"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment \"tiller-deploy\" successfully rolled out\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/tiller-deploy -n kube-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cordon off loadtest nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                                                STATUS   ROLES    AGE     VERSION\r\n",
      "gke-standard-cluster-1-default-pool-88be49f0-5533   Ready    <none>   8m54s   v1.12.8-gke.10\r\n",
      "gke-standard-cluster-1-default-pool-88be49f0-gbtg   Ready    <none>   8m53s   v1.12.8-gke.10\r\n",
      "gke-standard-cluster-1-default-pool-88be49f0-k4js   Ready    <none>   9m2s    v1.12.8-gke.10\r\n",
      "gke-standard-cluster-1-default-pool-88be49f0-qhvj   Ready    <none>   8m53s   v1.12.8-gke.10\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl get nodes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We cordon off first 3 nodes so seldon-core and the model will not be deployed on the 1 remaining node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "node/gke-standard-cluster-1-default-pool-88be49f0-5533 cordoned\n",
      "node/gke-standard-cluster-1-default-pool-88be49f0-gbtg cordoned\n",
      "node/gke-standard-cluster-1-default-pool-88be49f0-k4js cordoned\n"
     ]
    }
   ],
   "source": [
    "!kubectl cordon $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')\n",
    "!kubectl cordon $(kubectl get nodes -o jsonpath='{.items[1].metadata.name}')\n",
    "!kubectl cordon $(kubectl get nodes -o jsonpath='{.items[2].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Label the nodes so they can be used by locust."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "node/gke-standard-cluster-1-default-pool-88be49f0-5533 labeled\n",
      "node/gke-standard-cluster-1-default-pool-88be49f0-gbtg labeled\n",
      "node/gke-standard-cluster-1-default-pool-88be49f0-k4js labeled\n"
     ]
    }
   ],
   "source": [
    "!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') role=locust\n",
    "!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[1].metadata.name}') role=locust\n",
    "!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[2].metadata.name}') role=locust"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start seldon-core"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core\n",
      "E0625 13:28:54.390314    3430 portforward.go:363] error copying from remote stream to local connection: readfrom tcp4 127.0.0.1:41235->127.0.0.1:33400: write tcp4 127.0.0.1:41235->127.0.0.1:33400: write: broken pipe\n",
      "LAST DEPLOYED: Tue Jun 25 13:28:53 2019\n",
      "NAMESPACE: seldon-system\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/ClusterRole\n",
      "NAME                          AGE\n",
      "seldon-operator-manager-role  1s\n",
      "\n",
      "==> v1/ClusterRoleBinding\n",
      "NAME                                 AGE\n",
      "seldon-operator-manager-rolebinding  1s\n",
      "\n",
      "==> v1/ConfigMap\n",
      "NAME                     DATA  AGE\n",
      "seldon-spartakus-config  3     1s\n",
      "\n",
      "==> v1/Pod(related)\n",
      "NAME                                         READY  STATUS             RESTARTS  AGE\n",
      "seldon-operator-controller-manager-0         0/1    ContainerCreating  0         0s\n",
      "seldon-spartakus-volunteer-6954cffb89-dzqmv  0/1    ContainerCreating  0         0s\n",
      "\n",
      "==> v1/Secret\n",
      "NAME                                   TYPE    DATA  AGE\n",
      "seldon-operator-webhook-server-secret  Opaque  0     1s\n",
      "\n",
      "==> v1/Service\n",
      "NAME                                        TYPE       CLUSTER-IP  EXTERNAL-IP  PORT(S)  AGE\n",
      "seldon-operator-controller-manager-service  ClusterIP  10.0.26.77  <none>       443/TCP  0s\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME                              SECRETS  AGE\n",
      "seldon-core-seldon-core-operator  1        1s\n",
      "seldon-spartakus-volunteer        1        1s\n",
      "\n",
      "==> v1/StatefulSet\n",
      "NAME                                READY  AGE\n",
      "seldon-operator-controller-manager  0/1    0s\n",
      "\n",
      "==> v1beta1/ClusterRole\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  1s\n",
      "\n",
      "==> v1beta1/ClusterRoleBinding\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  0s\n",
      "\n",
      "==> v1beta1/CustomResourceDefinition\n",
      "NAME                                         AGE\n",
      "seldondeployments.machinelearning.seldon.io  1s\n",
      "\n",
      "==> v1beta1/Deployment\n",
      "NAME                        READY  UP-TO-DATE  AVAILABLE  AGE\n",
      "seldon-spartakus-volunteer  0/1    1           0          0s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "NOTES: TODO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true   --namespace seldon-system    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for 1 pods to be ready...\n",
      "partitioned roll out complete: 1 new pods have been updated...\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/seldon-controller-manager -n seldon-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Stub Deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "    \u001b[34;01m\"apiVersion\"\u001b[39;49;00m: \u001b[33m\"machinelearning.seldon.io/v1alpha2\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"kind\"\u001b[39;49;00m: \u001b[33m\"SeldonDeployment\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"metadata\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"labels\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"app\"\u001b[39;49;00m: \u001b[33m\"seldon\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"seldon-core-loadtest\"\u001b[39;49;00m\r\n",
      "    },\r\n",
      "    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"project_name\"\u001b[39;49;00m: \u001b[33m\"loadtest\"\u001b[39;49;00m,\r\n",
      "            \u001b[34;01m\"deployment_version\"\u001b[39;49;00m: \u001b[33m\"v1\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"loadtest\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_key\"\u001b[39;49;00m: \u001b[33m\"oauth-key\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_secret\"\u001b[39;49;00m: \u001b[33m\"oauth-secret\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"predictors\"\u001b[39;49;00m: [\r\n",
      "            {\r\n",
      "                \u001b[34;01m\"componentSpecs\"\u001b[39;49;00m: [{\r\n",
      "                    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "                        \u001b[34;01m\"containers\"\u001b[39;49;00m: [\r\n",
      "                        ],\r\n",
      "                        \u001b[34;01m\"terminationGracePeriodSeconds\"\u001b[39;49;00m: \u001b[34m20\u001b[39;49;00m\r\n",
      "                    }\r\n",
      "                }],\r\n",
      "                \u001b[34;01m\"graph\"\u001b[39;49;00m: {\r\n",
      "                    \u001b[34;01m\"children\"\u001b[39;49;00m: [],\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"stub\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"endpoint\"\u001b[39;49;00m: {\r\n",
      "\t\t\t\u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"REST\"\u001b[39;49;00m\r\n",
      "\t\t    },\r\n",
      "                    \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"MODEL\"\u001b[39;49;00m,\r\n",
      "\t\t    \u001b[34;01m\"implementation\"\u001b[39;49;00m: \u001b[33m\"SIMPLE_MODEL\"\u001b[39;49;00m\r\n",
      "                },\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"loadtest\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"replicas\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\r\n",
      "\t\t\u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "\t\t    \u001b[34;01m\"predictor_version\"\u001b[39;49;00m : \u001b[33m\"v1\"\u001b[39;49;00m\r\n",
      "\t\t}\r\n",
      "            }\r\n",
      "        ]\r\n",
      "    }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize resources/loadtest_simple_model.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/seldon-core-loadtest created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/loadtest_simple_model.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for deployment to be running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment \"loadtest-loadtest-9eecb7d\" successfully rolled out\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deployment.apps/loadtest-loadtest-9eecb7d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run benchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncorden the first 3 nodes so they can be used to schedule locust"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "node/gke-standard-cluster-1-default-pool-88be49f0-5533 uncordoned\n",
      "node/gke-standard-cluster-1-default-pool-88be49f0-gbtg uncordoned\n",
      "node/gke-standard-cluster-1-default-pool-88be49f0-k4js uncordoned\n"
     ]
    }
   ],
   "source": [
    "!kubectl uncordon $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')\n",
    "!kubectl uncordon $(kubectl get nodes -o jsonpath='{.items[1].metadata.name}')\n",
    "!kubectl uncordon $(kubectl get nodes -o jsonpath='{.items[2].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## gRPC\n",
    "Start locust load test for gRPC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   loadtest\n",
      "LAST DEPLOYED: Tue Jun 25 13:31:19 2019\n",
      "NAMESPACE: default\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/Pod(related)\n",
      "NAME                   READY  STATUS             RESTARTS  AGE\n",
      "locust-master-1-qxgr6  0/1    ContainerCreating  0         1s\n",
      "locust-slave-1-25sgw   0/1    Pending            0         0s\n",
      "locust-slave-1-2x4dg   0/1    Pending            0         0s\n",
      "locust-slave-1-64r2t   0/1    Pending            0         0s\n",
      "locust-slave-1-968hb   0/1    Pending            0         0s\n",
      "locust-slave-1-bm47q   0/1    ContainerCreating  0         1s\n",
      "locust-slave-1-cskhh   0/1    ContainerCreating  0         1s\n",
      "locust-slave-1-lrspz   0/1    Pending            0         0s\n",
      "locust-slave-1-m24xl   0/1    Pending            0         0s\n",
      "locust-slave-1-n4499   0/1    ContainerCreating  0         1s\n",
      "locust-slave-1-p59d5   0/1    Pending            0         1s\n",
      "locust-slave-1-qddf4   0/1    Pending            0         0s\n",
      "locust-slave-1-qq44k   0/1    ContainerCreating  0         1s\n",
      "locust-slave-1-rvjpm   0/1    ContainerCreating  0         1s\n",
      "locust-slave-1-t766z   0/1    ContainerCreating  0         1s\n",
      "locust-slave-1-vss82   0/1    Pending            0         0s\n",
      "\n",
      "==> v1/ReplicationController\n",
      "NAME             DESIRED  CURRENT  READY  AGE\n",
      "locust-master-1  1        1        0      1s\n",
      "locust-slave-1   64       0        0      1s\n",
      "\n",
      "==> v1/Service\n",
      "NAME             TYPE      CLUSTER-IP   EXTERNAL-IP  PORT(S)                                       AGE\n",
      "locust-master-1  NodePort  10.0.26.104  <none>       5557:31734/TCP,5558:32446/TCP,8089:31850/TCP  1s\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../helm-charts/seldon-core-loadtesting --name loadtest  \\\n",
    "    --set locust.host=loadtest-seldon-core-loadtest:5001 \\\n",
    "    --set locust.script=predict_grpc_locust.py \\\n",
    "    --set oauth.enabled=false \\\n",
    "    --set oauth.key=oauth-key \\\n",
    "    --set oauth.secret=oauth-secret \\\n",
    "    --set locust.hatchRate=1 \\\n",
    "    --set locust.clients=256 \\\n",
    "    --set loadtest.sendFeedback=0 \\\n",
    "    --set locust.minWait=0 \\\n",
    "    --set locust.maxWait=0 \\\n",
    "    --set replicaCount=64 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To download stats use \n",
    "\n",
    "```bash\n",
    "if [ \"$#\" -ne 2 ]; then\n",
    "    echo \"Illegal number of parameters: <experiment> <rest|grpc>\"\n",
    "fi\n",
    "\n",
    "EXPERIMENT=$1\n",
    "TYPE=$2\n",
    "\n",
    "MASTER=`kubectl get pod -l name=locust-master-1 -o jsonpath='{.items[0].metadata.name}'`\n",
    "\n",
    "kubectl cp ${MASTER}:stats_distribution.csv ${EXPERIMENT}_${TYPE}_stats_distribution.csv\n",
    "kubectl cp ${MASTER}:stats_requests.csv ${EXPERIMENT}_${TYPE}_stats_requests.csv\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get live stats by viewing the logs of the locust master"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " grpc loadtest-seldon-core-loadtest:5001                      13998858     0(0.00%)      10       0     525  |       9 5800.80\r\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------\r\n",
      " Total                                                        13998858     0(0.00%)                                    5800.80\r\n",
      "\r\n",
      " Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s\r\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------\r\n",
      " grpc loadtest-seldon-core-loadtest:5001                      14013158     0(0.00%)      10       0     525  |       9 5729.30\r\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------\r\n",
      " Total                                                        14013158     0(0.00%)                                    5729.30\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl logs $(kubectl get pod -l name=locust-master-1 -o jsonpath='{.items[0].metadata.name}') --tail=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "release \"loadtest\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!helm delete loadtest --purge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## REST \n",
    "Run REST benchmark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   loadtest\n",
      "LAST DEPLOYED: Tue Jun 25 14:14:35 2019\n",
      "NAMESPACE: default\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/Pod(related)\n",
      "NAME                   READY  STATUS             RESTARTS  AGE\n",
      "locust-master-1-6dp2w  0/1    Pending            0         0s\n",
      "locust-slave-1-2s5d4   0/1    Pending            0         0s\n",
      "locust-slave-1-4rqb9   0/1    Pending            0         0s\n",
      "locust-slave-1-5b5tm   0/1    Pending            0         0s\n",
      "locust-slave-1-5bnsn   0/1    Pending            0         0s\n",
      "locust-slave-1-769zb   0/1    Pending            0         0s\n",
      "locust-slave-1-7f468   0/1    Pending            0         0s\n",
      "locust-slave-1-8f2ng   0/1    ContainerCreating  0         0s\n",
      "locust-slave-1-8vzd6   0/1    Pending            0         0s\n",
      "locust-slave-1-b7lh2   0/1    ContainerCreating  0         0s\n",
      "locust-slave-1-dkpp4   0/1    Pending            0         0s\n",
      "locust-slave-1-fxjp6   0/1    Pending            0         0s\n",
      "locust-slave-1-gl4jg   0/1    Pending            0         0s\n",
      "locust-slave-1-hvqzm   0/1    Pending            0         0s\n",
      "locust-slave-1-jlxwp   0/1    ContainerCreating  0         0s\n",
      "locust-slave-1-ldthx   0/1    Pending            0         0s\n",
      "locust-slave-1-lhn66   0/1    ContainerCreating  0         0s\n",
      "locust-slave-1-ls9xm   0/1    Pending            0         0s\n",
      "locust-slave-1-m9x64   0/1    Pending            0         0s\n",
      "locust-slave-1-mm6p9   0/1    Pending            0         0s\n",
      "locust-slave-1-nt4m5   0/1    Pending            0         0s\n",
      "locust-slave-1-qttx5   0/1    Pending            0         0s\n",
      "locust-slave-1-rbvsw   0/1    Pending            0         0s\n",
      "locust-slave-1-s5tt4   0/1    Pending            0         0s\n",
      "locust-slave-1-t9dch   0/1    Pending            0         0s\n",
      "\n",
      "==> v1/ReplicationController\n",
      "NAME             DESIRED  CURRENT  READY  AGE\n",
      "locust-master-1  1        1        0      0s\n",
      "locust-slave-1   64       0        0      0s\n",
      "\n",
      "==> v1/Service\n",
      "NAME             TYPE      CLUSTER-IP   EXTERNAL-IP  PORT(S)                                       AGE\n",
      "locust-master-1  NodePort  10.0.20.244  <none>       5557:32399/TCP,5558:30063/TCP,8089:32046/TCP  0s\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../helm-charts/seldon-core-loadtesting --name loadtest  \\\n",
    "    --set locust.host=http://loadtest-seldon-core-loadtest:8000 \\\n",
    "    --set oauth.enabled=false \\\n",
    "    --set oauth.key=oauth-key \\\n",
    "    --set oauth.secret=oauth-secret \\\n",
    "    --set locust.hatchRate=1 \\\n",
    "    --set locust.clients=256 \\\n",
    "    --set loadtest.sendFeedback=0 \\\n",
    "    --set locust.minWait=0 \\\n",
    "    --set locust.maxWait=0 \\\n",
    "    --set replicaCount=64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get stats as per gRPC and/or monitor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " POST predictions                                               93406     0(0.00%)      19       3     393  |      16 4083.30\r\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------\r\n",
      " Total                                                          93406     0(0.00%)                                    4083.30\r\n",
      "\r\n",
      " Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s\r\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------\r\n",
      " POST predictions                                              105895     0(0.00%)      19       3     393  |      16 4289.00\r\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------\r\n",
      " Total                                                         105895     0(0.00%)                                    4289.00\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl logs $(kubectl get pod -l name=locust-master-1 -o jsonpath='{.items[0].metadata.name}') --tail=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "release \"loadtest\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!helm delete loadtest --purge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "node/gke-standard-cluster-1-default-pool-88be49f0-5533 cordoned\n",
      "node/gke-standard-cluster-1-default-pool-88be49f0-gbtg cordoned\n",
      "node/gke-standard-cluster-1-default-pool-88be49f0-k4js cordoned\n"
     ]
    }
   ],
   "source": [
    "!kubectl cordon $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')\n",
    "!kubectl cordon $(kubectl get nodes -o jsonpath='{.items[1].metadata.name}')\n",
    "!kubectl cordon $(kubectl get nodes -o jsonpath='{.items[2].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tear Down"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"seldon-core-loadtest\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f resources/loadtest_simple_model.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "release \"seldon-core\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!helm delete seldon-core --purge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
